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Abstract 

This paper presents an evaluation of the work of the Commonwealth of Learning’s (COL) 
eLearning with International Organisations (eLIO) section. Participants in the investigation 
included a representative sample of the learners (N = 15), their supervisors (N = 5), and the COL 
staff, including all of the eLIO staff (N = 10). The methodology consisted of an examination of 
all relevant documents, interviews that formed a learning history, and a sample survey. The 
investigation concluded that the eLIO achieved its goal of developing a distance learning model, 
and it met or exceeded identified objectives, with a high degree of satisfaction expressed by all 
participants. This included teaching +2000 satisfied learners; partnering with eight international 
organizations; achieving a 62% female participation rate and a high completion rate (75%) in the 
courses provided; testing, piloting, and delivering two new elearning courses; conducting needs 
analyses; recruiting/training highly qualified tutors; monitoring; and using appropriate 
technologies. Shortcomings of the programmes include the lack of pre- and post-tests, little 
analysis of pricing structures, some unclear - instructions (a need for plain English), unclear 
copyright licensing, only very limited use of available OER software, and the absence of a 
succession plan for the manager. Based on the high level of satisfaction among all participants, it 
was recommended that the section maintain its present work and address these shortcomings. 

Keywords: Distance education; open learning; e-learning; online learning; elearning 
management 

Commonwealth of Learning’s E-Learning for International 

Organisations Section 

The Commonwealth of Learning (COL) is an intergovernmental organisation created by the 
Commonwealth Heads of Government to encourage the development and sharing of open 
learning and distance education knowledge, resources, and technologies. The COL eLearning for 
International Organisations (eLIO) section provides technology-enhanced, customised training 
solutions for international organizations, such as the World Bank and UNESCO. Services include 
needs analyses, development of customized culture- and gender-sensitive training materials, and 
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course delivery at a distance using expert tutors. Many international organizations, aware of 
COL’s international reputation for distance education in the developing world, are contracting 
with the eLIO for their training needs. This paper provides an evaluation of this COL section and 
makes recommendations for improvements (Commonwealth of Learning, 2008h). 

Significance of the Investigation 

The increased availability of computers and the Internet has made possible the delivery of 
training to populations, particularly those in developing countries who previously could only be 
reached by using traditional classroom-based teaching approaches or by using often ineffective 
correspondence courses, which have high dropout rates. In order to ensure the effective adoption 
of these new technologies for learning, open access to case studies and other investigations of 
these distance education initiatives is important. These can serve as useful guides to others who 
are planning similar or complementary implementations. Knowledge of the research on 
successful or even unsuccessful training practices can help others to better plan their own systems 
(Cruickshank, 1990). 

The rapid advance of technology combined with the ever-increasing need for skilled professionals 
with technological and English language skills must be addressed. How to reach these 
professionals in widely dispersed international organizations, especially employees in developing 
countries, remains a serious concern. As researchers, we have an obligation to study programme 
implementations that address this problem and to provide an understanding of the difficulties and 
of the successes of the real-world implementations. 

For example, distance educators must evaluate the technologies that they choose for delivery. 
Which technologies are appropriate given the target population and their degree of access to 
computers and the Internet? As far back as 1983, Clark determined that the technology was 
neutral, conferring no learning benefits regardless of the media used. 

More closely related to this investigation is an examination of the role of the administration in an 
implementation project and in the training system as a whole. In recent times there has been a 
(perhaps necessary) focus on the learner as the centre of the process. This has often led to a de- 
emphasis on the training system as a whole and on the part that it plays. This study focuses on the 
administration of the training system. 

As a case study, this investigation may shed some light on what is needed for a successful 
implementation of distance training internationally. Flyberg (2006) reminds us that our 
understanding may be strengthened by the execution of a greater number of good case studies. 
Mathur and Oliver’s (2007) case study of an international blended learning programme for 
developing countries sheds light on the need for technological knowledge and also on the 
problems and pitfalls of collaboration between organizations. 

Oliveira, Cortimiglia, and Marques’ (2003) case study of a data technology distance education 
course implementation looked at the technology requirements, financial results, and market 
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possibilities. They concluded that their web course was capable of reaching remote areas and the 
instructional contents were capable of delivering professional qualifications. 

Hills (1999), in her case study of Lloyd’s bank training, focused on the organizational culture and 
its effect on training. She concluded that a learning culture is needed for the exploitation of 
multimedia for training. A faculty learning community was the subject of Long, Janas, and Kay, 
et al.’s case study (2009) of a small college online learning implementation. 

Failed implementations are also worthy of our attention, especially when the implementation is 
similar to one under investigation. Nchindila (2007) investigated the conditions for the success of 
the writing process in English using online mentoring in the workplace, concluding that in this 
case study the implementation was evaluated as not successful, based on questionnaires for the 
students and mentors. 

Outcomes and Outputs and Related Success Indicators 

The original outcomes and outputs were as stated in Table 1. 

Activities Implemented 

The eLIO team developed and delivered technology -mediated distance learning programmes with 
international organizations in different regions around the world. They formed partnerships with 
international organizations in order to accomplish this. The learning materials were developed 
and customized for each organization to promote the learning of relevant occupational skills, 
based on an analysis of the learners’ needs and on the capacity of the technological infrastructure 
available. Qualified tutors were engaged and trained in order to facilitate the delivery of the 
programmes. Seven international partners participated as partners in eLIO programmes. 
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Table 1 

Planned Outcomes and Outputs 


Outcomes 

• Development and delivery of technology-mediated distance learning programmes with international 
organisations to enable access to learning by a dispersed and diverse international workforce. 

• By 2009, at least 6 major international organisations active in delivering the Millennium 
Development Goals (MDGs) have formed a partnership with COL in customising gender- and 
culture-sensitive workplace eLearning programmes. 

• An additional 2,000 adult learners (a 6 to 4 ratio of women and men) will be trained by 
2009. 

• At least two new contextually relevant sets of ODL/ICT-based learning materials for 
occupational skills development created and each is taken up by at least two major 
institutions. 

• Did the COL eLIO achieve its goals through the activities posited? Activities: 

• Actions to achieve the outputs 

1. Analyse and understand the learners’ needs, learning objectives and outcomes, the 
learning environment and technology infrastructure of the organizations. 

2. Design courses and develop materials through customization. 

3. Test all training materials and delivery designs 

• Build responsive infrastructure and clear processes for course production and 
presentation. This includes 

• preparing storyboards/templates; 

• using robust, affordable and user-friendly technology and media; 

• recruiting and maintaining a cadre of trained and committed tutors; 

• using OERs software as far as possible. 

• Monitoring Data 

a. pilot evaluation reports 

b. monthly progress reports 

c. tutor listservs® 

d. end of course evaluation feedback 

e. programme evaluation studies 


Outputs 

Success Indicators 

• number of participants +2000 by 2009 

• number of organizations served 6 

• number of regions served not stated 

• revenue self sustaining by 2009 


Limitations of the Study 

This plan is limited to an evaluation of the activities of the COL eLIO and does not address the 
wider issues associated with the COL organization. The evaluation is for this particular section 
only, and it is recognized that the external validity or generalisability is limited. Reliability is 
always an estimate. It was addressed by triangulating the data collected using at least three 
different programme case studies, documents, a survey, and interviews with a random sample of 
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participants, COL eLIO staff, and others using similar questions and interviewing techniques. 
Through this triangulation a hybrid was attempted between quantitative and qualitative methods 
(SeeLeedy, 1993). 

The validity of the evaluation conclusions, in this case, cannot be assured by a control group. 
There is often a multi-group threat to the validity of an evaluation when comparing different 
groups taking the same programme. As this evaluation did not compare the outcomes of the 
different groups, but rather considered the section’s activities overall, including the programmes 
with different organizations, this was not a problem. 

Method 

The evaluation was conducted using triangulated data collected from several different pilot and 
programme evaluations, published papers, a simple survey instrument, and other documents as 
well as interviews with a random sample of participants (student, partners, and tutors). The COL 
staff and a sampling of others were interviewed using similar questions and interviewing 
techniques. These included seven COL staff members, five tutors, three partners, and 10 learners. 
The interviews were collated into a learning history document. 

The information collected consisted of a large corpus of documents, including previous project 
evaluations supplied by the COL eLIO, the COL, and other relevant websites’ published articles. 
Documents were obtained directly from the eLIO and from Internet searches. The documentation 
was positive in the extreme with none pointing to serious or even minor criticisms. At first this 
seemed suspicious but was borne out in the triangulation with the interviews and questionnaires. 

In addition, interviews were conducted with the COL eLIO staff, other COL personnel, and a 
small sample of participants in the COL eLIO projects. A random sample of representatives from 
three of the eight participating organizations was contacted and interviewed. A random sample of 
tutors (five from a list of 20) was also contacted and interviewed. Thirty students were randomly 
selected and asked to participate in an online survey using Survey Monkey. Ten responded and 
one agreed to be interviewed by telephone. The random samples were elicited using the 
application available at Random.com. The participants in each group were given a number and 
then the numbers were input into the random.com application, which randomized the total and 
picked a sample. The participants contacted were sent a consent agreement. 

A list of questions was used as the main instrument for the interviews. Other supplementary 
questions were used as they emerged out of the discussion with participants. The analysis 
included an examination of the question responses from which data was collected. This data from 
the quantitative questionnaires was analysed statistically (See Nunan, 1992). 

These surveys were triangulated with the interviews and documentation. The reliance on a variety 
of sources in this investigation is supported by Yin (1994), who argued that one of the benefits of 
a case study is that it depends on multiple sources of data as evidence. 
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The interviews were conducted either by telephone or in a face-to-face format in June 2008. The 
information was collected, converted into a learning history format, and analysed for this report. 
A learning history methodology provides organisations with a special approach to learning from 
the experiences of the participants. In the right-hand column, relevant observations are described 
by the people who took part. The participants are quoted directly, and their statements are placed 
in a category defined by their role to increase anonymity. The researcher then attempts to weave 
the words into a coherent story based on the information provided, including facts, results, and 
assumptions of the participants. This methodology is based on approaches and techniques in a 
variety of fields, including organizational learning (See Roth & Kleiner, no date). 

This evaluation consists of a summative assessment of the outputs and achievements of the COL 
eLIO for formative input into planning future programme implementations as well as for 
expanding partnerships and for increasing the efficiency and effectiveness of the sector. The key 
performance indicators - output and outcomes - have been identified from COL documents and 
in consultation with the COL eLIO representative. These include information on the 
organizations served and on the participants in the programmes delivered (See Table 1). 
Appropriate documentation and access to interviewees was made available to the evaluator, who 
developed a strategy for analysing the data provided. Indicators included data on the growth of 
the initiative (e.g., number of courses created and delivered, number of learners, learner 
achievement, engagement of technology, lessons learned, threats & opportunities, expansion, 
gender equity, etc.). 

This evaluation can be used by COL to inform its planning processes and to report on activities to 
COL stakeholders. 


Findings 

A 6R assessment criteria (relevance, results, reach, regions, resources, and relationships) was 
used. These criteria were triangulated with the documentation and survey results. 

Relevance 

There was a consensus among the COL staff that the eLIO is marginalized or “peripheral” 
primarily because it is the only section of COL that has been mandated to be self-supporting. 
They note that the eLIO, in fact, contributes significantly to the core mandate of COL as 
expressed in the COL Three Year Plan (Commonwealth of Learning, 2006). The eLIO’s 
relevance to COL is something more than revenue generation. There is a strong alignment of its 
goals in serving international organizations with those of COL in increasing capacity and 
demonstrating the effectiveness of open and distance learning (ODL). It strives to serve as a 
successful example of the use of technology for ODL in developing countries, including the 
Commonwealth. The section is responsible for adding to the reputation of COL as a professional 
organization that serves the developing world by using the best approaches and techniques in DE. 
As an organisation that works with governments at the policy level for human resource 
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development, COL’s reputation is enhanced when the eLIO, at the grassroots level, demonstrates 
the viability, effectiveness, and sustainability of international ODL. As one interviewee noted, 
COL “needs something to crow about.” 

Unexpected Outcomes 

The eLIO has also demonstrated its relevance by its actions because other opportunities have 
come to COL as a result of eLIO’s reputation for delivering ODL. UNICEF is interested in a 
wider opportunity for COL in HR development (possibly $1.9 million) along with a major 
capacity building project for South African civil servants. These opportunities have come to 
COL by word-of-mouth. Without advertising, the eLIO programmes have built viable 
relationships and have increased COL’s visibility among international organisations, particularly 
major funders. 


Results 

The results to be measured include in the first instance the stated intended results of the eLIO 
section as per the COL strategic plan. They are listed below along with information on the 
success of each. 

Goal 

Development and delivery of technology-mediated distance learning programmes with 
international organisations to enable access to learning by a dispersed and diverse international 
workforce. 

Results 

Successful: The eLIO developed and delivered several distance learning programmes to 
international organizations with widely dispersed and diverse international workforces. 

Goal 

By 2009, at least six major international organisations active in delivering the Millennium 
Development Goals (MDGs) have formed a partnership with COL in customising gender- and 
culture-sensitive workplace eLearning programmes. 

Results 

The eLIO surpassed this goal, partnering with eight international organizations active in 
delivering e-learning that support MDGs. These have been generally described as being sensitive 
to gender and cultural issues in the workplace. The following organizations participated as 
partners with eLIO: 

1. World Health Organisation, 

2. UN High Commissioner for Refugees, 

3. International Labour Organisation, 

4. International Federation of the Red Cross and Red Crescent Societies, 

5. The Joint UN co-sponsorships for HIV/AIDS, 
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6. World Bank, 

7. UNICEF, India Country Office, 

8. Office of Internal Oversight Services (OIOS). 


Goal 

An additional 2,000 adult learners (a 6 to 4 ratio of women and men) will be trained by 2009. 
Results 

1,766 adult learners (62% female) have received training from the eLIO section. This number 
increased to more than 3000 in 2009. 

Goal 

At least two new contextually relevant sets of ODL/ICT-based learning materials for occupational 
skills development created and each is taken up by at least two major institutions. 

Results 

Two new elearning courses have been developed, adapting the materials and contextualizing 
them to different organizations. 

Goal 

Did the COL eLIO achieve its goals through the activities posited? 

a. Analyse and understand the learners’ needs, the learning objectives and outcomes, the 
learning environment, and the technology infrastructure of the organizations 

Results 

Needs analyses were conducted with the partner organisations, taking into account the 
learning environment and the technological infrastructure. Adjustments to the courses 
were made based on these analyses; for example, deadlines and examination schedules 
were changed, CDRoms were used for distribution, and email was used rather than more 
complex applications on the Internet. 

b. Design courses and develop materials through customization 
Results 

Six courses were adapted and delivered specifically for each organization, taking into 
account their different cultures and learning environments. All partners interviewed 
agreed that this was done very well and according to their needs. 

c. Test all training materials and delivery designs 

i. build responsive infrastructure and clear processes for course 
production and presentation 

ii. prepare storyboards/templates 
Results 

One partner reported on the difference between her organization’s 
approach to instructional design and that of eLIO, specifically 
referring to eLIO’s change in approach to accommodate her 
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organization’s wishes. This included adopting storyboards and 
templates as part of the learning design process. 

iii. use robust, affordable, and user-friendly technology and media 
Results 

The eLIO course designers deliberately focused on low technology 
solutions in order to ensure that the course would be robust, user- 
friendly, affordable, and, most importantly, accessible by the learners 
dispersed around the world in developing countries. This included 
the distribution of CDRoms for the course materials and the use of 
simple email for communications. 

iv. recruit and maintain a cadre of trained and committed tutors 
Results 

The eLIO recruited and maintained a cadre of trained and committed 
tutors. The partners and students unanimously agreed that the tutors 
were indispensable and the most important part of the programme. 

v. use OER software as far as possible 
Results 

No open educational resource (OER) software was used. The project 
was limited by the proprietary software available in the partner 
organizations. For example, MS Office, including MS Word, was a 
standard that was known and used by the participants. 


d. Monitoring Data 

i. pilot evaluation reports 
Results 

A selection of pilot evaluation reports was made available for this 
study (Commonwealth of Learning, 2003; Murphy, 2003; Ruhe, 
2002; Shale, 2003) 

ii. monthly progress reports 
Results 

Monthly progress reports supplied by the tutors in a standard format 
were a regular occurrence. A random sampling of these reports was 
provided to the evaluator. 

iii. tutor listservs® 

Results 

Tutors were able to communicate with their students, the 
administration, and the administrators using listservs. In addition a 
web conferencing application was made available for discussion 
groups. 

iv. end of course evaluation feedback 
Results 

Learners were asked in the evaluations to provide their views on the 
course, their estimation of its success, and their attitude towards it. 
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They were satisfied with their progress and very positive in their 
attitude towards the course and the tutors in particular. However, 
other than some very tenuous pre- and post-testing, there was no 
reliable testing of the achievement level of the learners. There are 
problems associated with administering such pre- and post-tests in a 
distributed environment. Nevertheless, as was pointed out by some 
of the participants, such a practical test should be an integral part of 
the programmes in order to show learners, partners, tutors, 
administrators, and outsiders that real progress in achievement levels 
was made through the course, 
v. programme evaluation studies 
Results 

Programme evaluation studies were conducted and acted upon 
(Dunlop, 2005; John, 2005; Walker & Dzakiria, 2005; Wilson, 2006 
& 2007). 

The following questions were also addressed. 

Does the existing data gathering mechanism adequately cover the necessary information for 
analysis and reporting? Where are the gaps? 

The evaluations of both the pilots and the full courses were conducted in a professional manner. 
The learner feedback, although difficult to arrange, was adequate for analysis with a return rate of 
30% to 40%. As previously stated, the only real gap in the learner evaluations is in the need for 
pre- and post- tests of achievement. 

The time allotment by COL eLIO staff was set at 2% for initiative monitoring and evaluation. The 
staff budget was capped at 25% of the revenue generated. Was this goal achieved? By how 
much? 

This refers to funding allotted for program monitoring and evaluation and it was achieved. 
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Table 2 

Output Summary 


Output summary 
2007 - 2008 fiscal year 

o Number of participants in programmes: 1,085 
o Number of organizations served: 8 
o Number of regions served: 4 

o Revenue: ~$700 000 Costs:~$500,000; Difference: $200,000 (28%) 
* $578,032 (to Feb. 29, 2008) 

2006 - 2007 fiscal year 

o Number of participants in programmes: 681 
o Number of organizations served: 6 
o Number of regions served: 4 

o Revenue: $544,193 Costs: -$450,000; Difference: $100,000 (18%) 
o 138 countries; 28 Commonwealth countries 

o 15 of the least developed countries 


Besides these outcomes and outputs that were the goals of the eLIO as set out in the COL Three 
Year Plan , there were other results that need to be taken into account in any reasonable evaluation 
of the section. Foremost is the fact that more than 2,000 learners became more confident in their 
English writing skills. Many of these learners are either in Commonwealth countries (28), or they 
liaise with officials in Commonwealth countries. The student survey results showed that learners 
overwhelmingly felt that their writing and analytical skills had improved and that the course 
content was sufficient, clear and easy to understand, and appropriate for their organizations. They 
were particularly emphatic about the value of the tutorial system. 

Unexpected Opportunities 

Certainly, the unexpected opportunities with UNICEF and South Africa are among the unplanned 
results that can be used to further validate the section’s work. The very high esteem for COL 
expressed by the partners is also important as their word-of-mouth advertising not only led to 
more partners within the UN system, but also to the above-named opportunities. There is real 
evidence of the success of the programme when partners renew their contracts while boasting of 
success to the others (“thrilled we found them!”). The customized curriculum for each 
organisation along with the personalized service was highly appreciated. The tutor system was 
also an important result that may not be apparent immediately. Tutors’ empathy, flexibility, and 
personal skills, combined with a tight organizational structure, were critical components of the 
success of the programme. And it can be considered to be a key result of the programme itself 
when taken as a whole. 
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Another significant result for the benefit of the partner organizations, which might also be of 
service to COL if implemented internally, was the programme’s emphasis on standardizing 
communication practices around plain English writing. Partners felt that this helped them to 
standardize core functions among the learners, who operate using disparate cultural norms. More 
widely, as the courses spread to the different international agencies, it has become “a first,” one 
of the rare examples of sharing resources among the different UN organizations. 

On the negative side, the learners, the partners, and the tutors, while valuing the content highly, 
feel that there is room for improvement; for example, there is a need for clear section markers, a 
need for more contextualization, and a lack of variety in the examples. Tutors felt that the 
materials should be edited to reflect good practice in plain English. 

Reach 

Partners are convinced that online courses are the only viable way that they can reach their 
distributed staff for training. They particularly emphasise this need for ODL in developing 
countries and for staff who are traveling or who have fragmented work assignments. In this 
regard, they support the distribution of the courses on CD because of the erratic nature of Internet 
connectivity in many countries. 

Tutors see a need for organizations to be more proactive in supporting ODL as a means of 
extending their training reach. They need to develop a culture of using DE. Tutors could be used 
to train those within the organisation in ODL skills. In addition, more courses on a wide range of 
training subjects could be developed and delivered. 

Internally, staff members refer to the need for other COL sections to benefit from the expertise 
and skills of those in the eLIO section. They feel that the section should be reaching out to other 
COL sections that could use its expertise in capacity-building and human resource development. 
Reaching out can also be achieved successfully by making the most effective use of new 
technologies as they emerge. The section should experiment more with novel applications as 
access to them becomes more common among the partners. 

Regions 

Courses were delivered in four regions of the Commonwealth (Africa, South Asia, the Pacific, 
and the Caribbean). As previously mentioned, among the most significant developments arising 
from the activities of the eLIO is the opportunity for a contract to deliver training to civil servants 
in South Africa. Using Canadian tutors to train South Africans as ODL tutors could strengthen 
COL’s core mandate for capacity-building. The different regions could also be strengthened by 
using the technology to create regional groupings of learners, so they can engage in peer-to-peer 
learning and teaching. 
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Resources 

The ambiguous position of the eLIO was referred to previously. COL’s emphasis on extra 
budgetary financing has resulted in the section being perceived as different from other COL 
sections. The eLIO may not be considered as “core” by the COL board particularly because it 
serves many non-Commonwealth countries. The eLIO is perhaps both core and extraneous and is 
more useful to COL than it might be otherwise because from within the organisation it is able to 
provide an opening or a window to other international organizations involved in complementary 
activities. Therefore, eLIO can be seen as a resource. 

The eLIO’s emphasis on using simple technologies, while important for promoting access and 
supporting students, may not be the best strategy in the management and administration of its 
programmes. Staff members use applications like Survey Monkey for their surveys, so they are 
taking some advantage of the technologies available. The recent adoption of Basecamp for 
computer conferencing is another example. However, there are other open source applications 
that staff might experiment with to improve the management and the efficiency of the learning 
process. This experimentation with technologies could form part of an extended mandate for the 
section in the research and development of DE. For example, a customer relationship 
management (CRM) application may help to improve learner support and administrative 
efficiencies. Open source applications do exist (e.g. SugarCRM, Vtiger, Campware). 

The Learning Manager 

Many of the respondents commented specifically on the professionalism and competence of the 
eLIO staff, making specific reference to the learning manager. She was commended for turning 
her section from an “ugly duckling” to a “swan.” Others refer to the section as being very 
“person-based.” Others say that she runs a “tight ship” as an excellent organizer and 
administrator. Some wonder if the section would have a future if this particular' learning manager 
was no longer running it, even suggesting that COL would have to review the section and perhaps 
re-organize it if she leaves. 


Discussion 

From the beginning, there has been debate about the role of the eFIO in promoting the 
organization, developing partnerships, raising funds, and providing support for ODF. An 
important development in the debate has been the evolving understanding of the section’s role in 
supporting COF’s overall policy direction and, hence, the section’s alignment with the 
organization’s goals. For example, the eFIO’s efforts in delivering training to UN agencies have 
resulted in opportunities from UNICEF and possibly from the South African government for 
other COF sections. These opportunities can relate directly to COF’s core goal of capacity- 
building in the Commonwealth. 

After the last evaluation, COF priorities moved in the direction of extra budgetary funding for the 
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eLIO. In the past, the role of COL has swung between supporting demonstration projects and 
providing limited funding of research. In the present three-year plan, COL has settled into a 
preference for intergovernmental relations and policy. This very brief recounting of the shifting 
forces that have shaped COL serves at least to signal the twists and turns of both COL’s goals and 
management philosophy, which the eLIO learning manager has had to follow. Without an 
appreciation of these shifts, evaluating the successes and failures of the eLIO programme could 
have been a very frustrating and puzzling enterprise. 

Conclusions 

The eLIO achieved its intended results, yielding significant benefits in its principal outcomes and 
outputs. It developed a model for potential application in a variety of different (but possible) 
organizational and educational settings or environments with some important additions to the 
stock of open learning pedagogical knowledge. Although not directly generalizable to other 
organizations, this case study and its conclusions and recommendations could be used by others 
in formulating implementation plans and developing strategies. It does add to the body of case 
studies and can be useful, as Flyvberg (2007) suggests. 

On the other hand, the section has not employed a consistent methodology for estimating and 
evaluating the benefits from its programmes. Evaluations tend to focus on attitudinal benefits 
regarding ODL and individual experiences, rather than taking into account the broader array of 
benefits (realized and otherwise) flowing from its grant work and other initiatives. Finally, the 
way in which the section’s programmes were organized and managed made a real difference to 
the benefits that were produced by the initiatives undertaken. 

To assess the benefits of the first iterations of the elLO programmes, the section engaged 
evaluators to examine the programmes, to report on the benefits and problems encountered, and 
to make recommendations. These evaluations can be seen as mini case studies of each 
programme. The projects were similar and the evaluations tended to cluster around the same set 
of benefits and recommendations, for example stressing the high quality of the instructional 
materials and the importance of the tutor-learner relationship. There were sufficient programme 
evaluations to reliably inform the analytical framework and to draw reliable conclusions about the 
success or failure of the overall program. 

Perhaps the most difficult analytic problem is assigning to the section a proportion of the overall 
benefit of the programmes in a manner that properly reflects the section’s contribution to it. For 
example, how much of the partners’ and learners’ input was responsible for the overall success? 
There is no reliable way to accurately quantify the section’s contribution in most cases, and doing 
so remains a methodological challenge for the future; although, having a pre- and post-test of 
learner achievement should go a long way towards increasing the reliability of any evaluation. 
For the purposes of this study, the evaluator has attempted to specify in his analyses the role that 
the eFIO section has played. Based on this assessment a conservative judgment was used to 
characterize the section’s contribution for purposes of developing findings and recommendations. 
What cannot be dismissed in any analysis of the work of this section is the fact that all of the 


14 


A Case Study of an International E-Learning Training Division: Meeting Objectives 

McGreal 


stated goals were met and in some cases exceeded, coupled with an overwhelming degree of 
satisfaction expressed by all of the participants, including the learners, the partners, the tutors, the 
COL staff, and the eLIO section employees. 

Recommendations 

Taking into account the general level of satisfaction and increasing number of clients, the 
following recommendations are suggested: 

1. Keep doing what you are doing. 

2. Edit and revise the material to model plain English. 

3. Re-examine your pricing structure to maximize revenues as much as possible. 

4. Negotiate with partners either copyright control or an open license. 

5. Implement new technologies to support the learning management and administration 
(LMS, CRM). 

6 . Develop a succession plan for the section if the present manager leaves. 
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